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(54) A user interface 

(57) A user interface and a method for providing the 
user interface is disclosed. In the method a user is pro- 
vided with a view (11). A real woild object (20) that is 
visible in the view is then selected. A virtual object (22*) 
Is also displayed for the user, said virtual object being 
associated with an action. The user may select the ac- 



tion by moving the view (11) and the virtual object (22') 
relative to the object (20) such that the object and the 
virtual object become associated with each other. The 
user interface may comprise display means enabling 
the user to see the real worid object through the display 
means. The display means may comprise a head 
mounted display or a hand-held display. 
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Description 

[0001] The present invention relates to a user inter- 
face, and in particular, but not exclusively, to a user in- 
terface for interaction with a real world object. s 
[0002] A user may wish to interact with a device with- 
out e.g. touching a control button of the device. For ex- 
ample, a user may wish remotely control a device such 
as a domestic appliance, an office appliance, a vending 
machine, an entrance gate, and so on. Conventionally 
the control instructions have been provided by operating 
a control means provided either on the device or on a 
specific remote controller of the device. The interaction 
may also comprise other taslcs such as communication 
of information between the user and the device and so 
on. The sidlled person is familiar with the operation of 
such conventional user interfaces and thus these are 
not be discussed in any great detail herein. 
[0003] it has been proposed that the user interface 
could be provided based on the so called augmented 
reality (AR) an'angements. The temi augmented reality 
refers generally to arrangements wherein a real-world 
view and a computer generated view can be combined. 
The combining may be seamless so that the view per- 
ceived by the user through the display equipment is a 
combination of objects in a real-world image and com- 
puter-generated i.e. virtual objects. Conventionally the 
augmented reality has been used for enabling a user to 
receive information regarding the environment he/she 
is looking at. An augmented reality display equipment 
may comprise a head mounted display device. 
[0004] in a prior art approach a virtual object such as 
a menu or a text message may be shown to the user on 
the display. The user may then use an input device to 
interact with the virtual object. The input device may 
comprise an one-hand mouse, a scroller, a iceyboard, a 
joysticic or similar Icnown device. For example, the user 
may push a joystick upwards or downwards. This will 
then scroll the displayed menu up or down, respectively. 
The approach requires the user to use at least the other 
hand to operate the device. The coupling between the 
user input and the presentation device indirect and thus 
is it not provided in an intuitive or straightforward man- 
ner. 

[0005] in another prior art approach a set of virtual ob- 
jects (e.g. menus or a set of activation buttons) repre- 
senting possible actions is displayed to the user. The 
user may then move the virtual objects on the display 
so that the desired virtual object is located In a specific 
location (e.g. in the centre) of the field of view of the 
user. The user may then select the action by pressing a 
button or by some other input device. However, the 
method requires the eyes of the user to be focused to 
the virtual object when selecting the desired action. 
Thus the user may loose his/hers visual contact with the 
real world object. In other words, the user may not be 
able to see the real world object while selecting the ac- 
tion. 



[0006] The user may also use his/her hand to point to 
virtual objects that represent possible actions and are 
visible in the field of view of the user. The system com- 
putes which object was pointed at and after the compu- 
tations performs the specified action. A drawback of this 
type of arrangements is that the hand needs to be rec- 
ognised by the system or there needs to be at least a 
set of sensors to detect the position of the hand in a 
three dimensional (3D) space relative to the head of the 
user. IHands-free operation is by definition impossible. 
Using pointing gestures may cause hand injuries or oth- 
er accidents (e.g. pushing a coffee cup off from the ta- 
ble) when deeply focused to the augmented reality con- 
tent. Using pointing gestures may be socially unaccept- 
able in certain situations and may be a subject of differ- 
ent user preferences. For example, some people may 
not want to use gestures because they may fear that 
this coutd make them look strange. 
[0007] The user may also use a voice activation sys- 
tem forgiving commands. However, spoken commands 
may not always be used. For example, voice commands 
may be socially unacceptable in certain situations, such 
as in a theatre or other public places. Votoe commands 
may also not be desired in other circumstances wherein 
silence and/or secrecy is required, e.g. In certain police 
or military operations. 

[0008] The inventors have found that there may not 
be any satisfactory solution for a user to intuitively and 
conveniently interact with real worid objects in his/her 
environment The user may also wish to Interact with 
various real worid objects with embedded computing ca- 
pabilities such as domestic appliances with a Blue- 
tooth^*^ connectivity. A problem Is how casual Interac- 
tion, i.e. interaction which lasts only a substantially brief 
moment could be implemented. 
[0009] Embodiments of the present Invention aim to 
address one or several of the above problems. 
[0010] According to one aspect of the present inven- 
tion, there Is provided a method of providing a user In- 
terface, the method comprising: providing a user with a 
view; selecting an object that is visible in the view; dis- 
playing a virtual object for the user, said virtual object 
being associated with an action; and selecting the action 
by moving the view and the virtual object relative to the 
object such that the object and the virtual object become 
associated with each other. 

[001 1 ] According to another aspect of the present in- 
vention there is provided a user interface comprising 
display means, the display means being adapted for dis- 
playing a virtual object for a user and enabling the user 
to see a real worid object through the display means, 
wherein the user is enabled interact with the object by 
moving the display means relative to the real worid ob- 
ject such that said virtual object is associated with the 
real worid object. 

[001 2] In more specific forms of the invention the user 
is provided with a head mounted or a handheld display 
devrce comprising a see-through display. 



10 



15 



20 



25 



30 



35 



40 



45 



50 



55 



2 



3 



EP 1271 293 A2 



4 



[0013] The object may be elected by positioning It 
within a selection area of the view. The object may be 
selected automaticalty after the object has been held in 
said selection area tor a predefined period of time. The 
selection may also be triggered by the user. s 
[0014] The object may send a signal. The object may 
be recognised based on the signal. The object may be 
Identified based on the signal. 
[0015] The object may also be recognised by a cam- 
era means. The recognition and/or Identification of the 
object may be based on pattern or shape recognition. 
The pattern may comprise a barcode. 
[0016] The virtual object may comprise an area in the 
view. 

[0017] The association between the object and the 
virtual object is provided by aligning the objects with 
each other. 

[0018] An action may be selected after the associa- 
tion between the object and the virtual object has been 
maintained for a predefined period of time. The user 
may confinn that the object shall be subjected to an ac- 
tion that is indicated by said association between the 
object and the virtual object. 
[0019] The object may be subjected to at least one 
control operation In response to said selection of action. 
[0020] Infonnation associated with at least one pos- 
sible action may be communicated between the object 
and a control entity of the display means. Such informa- 
tion may also be stored in a control entity of the display 
means and/or communicated via a data network to a 
control entity of the display means. 
[0021 ] The embodiments of the invention provide the 
user with an easy way to interact with real-world objects. 
The embodiments provide a strong visual coupling be- 
tween a selected real-worid object and an object repre- 
senting an action the user wishes to be taken. The user 
may use subtle head movements or otherwise move the 
image window he/she sees relative to the object to ob- 
tain a desired effect. The interaction may not distract 
other people. A completely hands-free operation is pro- 
vided by some of the embodiments. The system Is also 
substantially easy to use. Relatively intuitive operation 
may be provided. If the user's environment is provided 
with a plurality of devices that can be tele-operated by 
the user the user may be provided with an easy way to 
control or otherwise Interact with these devices. 
[0022] For better understanding of the present Inven- 
tion, reference will now be made by way of example to 
the accompanying drawings in which: 

Figure 1 shows one embodiment of the present in- 
vention; 

Figures 2a and 2b show views as seen by a user of 
the Figure 1 device In accordance with an embodi- 
ment of the present invention; 
Figures 3a to 3d show a further embodiment of the 
present invention; 

Figure 4 is a flowchart illustrating the operation of 



one embodiment of the present Invention; and 
Rgure 5 shows a handheld device embodying the 
present invention. 

[0023] Reference Is made to schematic Figure 1 
which shows an embodiment of the present invention. 
In the embodiment a user 1 is provided with a head- 
mounted see-through display (IHMD) 10 and a control 
unit 2 for controlling the display. The head mounted dis- 
plays are known as such and are thus not described in 
any great detail herein. 

[0024] The display means is adapted to enable the us- 
er to experience augmented reality (AR). The display 
means is also adapted to provide the user with a user 
interface (Ul) for use in interaction with real worid ob- 
jects. 

[0025] An appropriate communication connection 3 Is 
provkied between the display unit 1 0 and the control unit 
2. The control unit may be attached by any appropriate 
manner to the body of the user. The user may also keep 
the control unit in his/hers pocket or held the unit in 
hand. According to a possibility control means are pro- 
vided in the head-mounted display apparatus. 
[0026] The user 1 wearing the head-mounted see- 
through display (HMD) 10 may see the real worid view 
he/she is looking at through the display screen. At the 
same time additional infomnation can be presented to 
the user 1 by means of at least one virtual object gen- 
erated by the control unit 2. For an example of the virtual 
objects, see e.g. Figure 2a. 

[0027] The additional information may associate with 
the view the user Is looking at or the environment the 
view relates to. As shown by Figure 4, the additional in- 
formation may also associate with control instructions 
the user is enabled to give for the real world object he/ 
she she Is looking at. The additional Information may 
also be any other information the user is interested in 
receiving at the same time he/she is looking through the 
head mounted see-through display 10. 
[0028] The user may be viewing through transparent 
display means an object with which he/she wishes to 
interact. Figure 2a shows a field of view or window 11 
the user 1 sees through the display means 10. A real 
world object 20 appears in the middle of the view 11. 
Four computer generated virtual objects 22 also appear 
in the field of view 11 . A virtual object 22 may comprise 
e.g. a cieariy visible selection area. The selection area 
is shown to have an oval shape in Figures 2a and 2b. it 
shall be appreciated that the virtual objects may com- 
prise any graphical logos, "icons" and so on. 
[0029] The user may use the virtual objects 22 for in- 
teraction with the real worid object 20. For example, the 
user may control a devk:e by selecting an action repre- 
sented by the virtual object. Possible implementations 
of the virtual objects will be explained In more detail after 
the following explanation of the principles of an embod- 
iment of the present invention. 
[0030] In accordance with an embodiment the user 1 
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may interact with real-world objects in the nearby envi- 
ronment of the user by first selecting a desired real world 
object 20. The selection may be accomplished by look- 
ing at the object 20 so that the object appears in a spe- 
cific point or area 21 of the view 11 . For example, the 
object may be positioned in a selection area located In 
the center of the view. Naturally, the selection area 21 
may be located In any location of the viewing window 1 1 . 
[0031] Although not necessaiy in ail applications, the 
object 20 may be adapted to facilitate recognition there- 
of by the augmented reality system. More particuiarty, 
the control unit 2 such as a wearable computer may rec- 
ognise when a real world object is visible in the field of 
view 11. To implement this the object 20 may contain, 
for example, means such as a directional RF transceiv- 
er, an infrared beacon, visual tags (e.g. barcodes or spe- 
cific pattems) and so on. The object 20 may also have 
a distinct visual appearance which allows it to be easily 
tracked and recognised e.g. by a camera or other de- 
tection means. 

[0032] I n addition , th e object 20 may be provided with 
means which allow communication between the aug- 
mented reality system and the object 20. For example, 
the object 20 and the control unit 2 may each be provid- 
ed with short range radio link modules 5. The commu- 
nication media between the modules may be based on 
the Bluetooth^^ protocol and transceiver modules. 
[0033] The recognition and the communication may 
be Implemented by using the same underlying technol- 
ogy. For example, the recognition and the communica- 
tion may both be based on use of infrared technology 
such as the ones based on the IrDA protocol or short 
range radio links such as the Bluetooth^^. 
[0034] The object 20 can be selected automatk:ally for 
interaction by keeping it in the selection area 21 for a 
specified period of time. For this purpose the control uni- 
ty may be provided with a timer function 7. The automat- 
ic operation provides the userwith hands free operation. 
[0035] An input device may be required in some ap- 
plteations. For example, the user may wish to have a 
possibility to manually confirm the selection of an object. 
However, a substantially simple input device may be 
enough. For example, the user may initiate the proce- 
dure and activate the user interface by pressing a con- 
trol button or operating other appropriate control means, 
e.g. votee activation means. The activation may be ac- 
complished while the object 20 appears to be in the se- 
lection area 21 . 

[0036] There are various alternatives for the imple- 
mentation of the control means. For example, a control 
means such as a control button or similar may be em- 
bedded in the clothing of the user or the control means 
may be provided by a hand held control device. The con- 
trol means may a!so be provided on the control unit. The 
user may also operate the user interface by using some 
specific gesture or muscle movement, or by a voice 
command and so on. 

[0037] Once the object 20 has been selected the sys- 



tem may indicate this for the user. For example, a suc- 
cessful selection may be indicated by visual, audible, or 
tactile signals. 

[0038] One or more possible control actions can be 
5 displayed to the user by appropriate visual virtual ob- 
jects 22 In the field of view of the user. The possible ac- 
tions may be presented by using graphical symbols. The 
symbols i.e. "icons" may comprise text, images (e.g. log- 
os), a combination of text and images and so on. The 
10 virtual objects 22 are refen'ed to in the following as ac- 
tion drop areas. 

[0039] The infomiation associated with the possible 
actions may reside in the augmented reality system. For 
example, the information may be stored in a database 

'5 4 of the control unit 2. At least a part of the required 
infomiation may also be downloaded from the real-world 
object via a communication media between the object 
and the user device. At least a part of the Infomnation 
may be downloaded from a remote data storage means 

20 such as a server. The downloading may occur via ap- 
propriate data network such e.g. the Internet, a local ar- 
ea network and so on. It is also possible that the required 
infomnation is obtained from several sources by using 
any combination of above techniques or by any other 

25 appropriate technique. 

[0040] After having selected the object 20 the user 1 
may move his/hers head (see the arrow in Figure 2b) so 
that the selected object 20 appears in association with 
an action drop area 22* in the field of view 11 . More par- 

30 ticularly, the possible actions (I.e. the drop areas 22) 
may be made visible in the view 11 after selection of the 
object. After this the user 1 can select the desired action 
by moving his/hers head so that the object 20 and a de- 
sired action drop area 22 become aligned. In Figure 2b 

3s an Indication of the selection is given by displaying the 
virtual object 22' such that it is larger than the other vir- 
tual objects 22. 

[0041] The control unit 2 may notice the selection of 
the action by tracking the real word object 20 and com- 

40 puting where the object 20 is located in the field of view 
11 . The selection of the action may be detected auto- 
matically once the control unit 2 notk:es that the real 
world object 20 is aligned with an action drop area 22. 
The timer 7 may be used for ensuring that an action drop 

4s area has not been selected accidentally. Alternatively, 
as with the selection area 21 , the user 1 may indicate 
the moment when the object 20 is aligned with a desired 
drop site 22 by using a button or some other control 
means. 

so [0042] In response to the indication the control unit 2 
resolves whk:h one of the action drop areas 22 is aligned 
with the object and initiates a corresponding action. 
Once the desired action is resolved the augmented re- 
ality system may carry out the desired action. This may 

55 involve communication with the real world object and/or 
with remote servers or any other action . The action shall 
be understood to refer to any action which may be ac- 
complished in response to an instruction from a user. 
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[0043] The system may be provided with means for 
tracking the movements of the user's head. This is for 
tracking the relative movement between the view 1 1 and 
the real-world object 20 and/or to detect if the real-world 
object 20 in the environment is aligned with a drop area 5 
22, Such tracking means as such are known and will not 
be described in any greater detail. It is suffteient to note 
that the tracking can be provided e.g. by means of an 
electronic compass, a gyroscope, by tracking an infra- 
red beacon associated with the object by means of a 
sensor-array, by using a camera to visually track the ob- 
ject or the infrared beacon associated with the object 
and so on. A combination of more than one tracking 
technique may also be used. 
[0044] Active tags I.e. tags which actively emit signals 
to the nearby environment may be used. For example, 
the tracking of the object 20 may be implemented by 
using an Infrared beacon attached to the real^orld ob- 
ject 20. The beacon may then be tracked by using an 
infrared sensitive camera. The beacon generation 
means may encode the identity of the object to the emit- 
ted signal. A drawback of using active tags may be that 
they may require replacement/recharging of batteries if 
the object are not connected e.g. a to power outlet or 
are not energy self-sufficient (e.g. powered by a sun- 
power, wind power and so on). 
[0045] It is also possible to use passive tags i.e. tags 
whk:h do not actively emit any signals to the nearby en- 
vironment. One way to implement tracking of the pas- 
sive tags is to use camera arrangement that is adapted 
to track an object. For example, the object may have a 
distinct marking, e.g. 2D barcodes or a specific pattem 
which helps to detect and track the object. Similarly, the 
object may have distinct visual appearance whk;h can 
be used to track it. 

[0046] i nf ormation based on which it is possible to vis- 
ually track an object (e.g. 2D barcode ID or visual ap- 
pearance infomiation) can be stored at the storage 
means 4 of the controller unit 2. The infomnatlon may 
also be mediated to the augmented reality system be- 
fore initiation of the tracking procedure. For example, 
the real-world object 20 may send the 2D barcode ID 
thereof or visual appearance infomriation to the aug- 
mented reality system by using a local short range radio 
link. The identity data may contain encoded address in- 
fomiation such as a universal resource locator (URL) 
which may then be used to obtain information from a 
data network, such as the Intemet. 
[0047] The augmented reality system may be enabled 
to use the tracking feature only when the user indicates 
a desire to interact with an object in the visual environ- 
ment of the user. In an automatic mode the augmented 
reality system may be activated only if a predefined real- 
world objects is detected to be present in the view. For 
example, the tracking should only be initiated when a 
short range radio link or an infrared beacon or other trig- 
gering event is detected. 

[0048] The system may distinct the various areas of 



the view 11 by means of different visual appearance of 
the different areas. For example, the selection area 21 
and/or the action drop areas can be displayed to be vis- 
ually different from each other in the view 11 . This may 
be accomplished e.g. by displaying a sem [transparent 
coloured or shaded areas 21 , 22 in the view 1 1 . The view 
11 may also be provided with a pointer element such as 
an an^ow or the like, said pointer element being an indi- 
cation of the area where the real worid object shall be 
placed. 

[0049] A reference is now also made Figures 3a to 3d 
which illustrate a more specific example of such inter- 
action and to the flowchart of Figure 4. More particularly, 
Figures 3a to 3d illustrate a specific example wherein 
the user interface is used to control a television set 30. 
[0050] A user provided with a head mounted display 
apparatus nnay move his/hers head so that the TV set 
30 is visible in the center area of his field of view (Figure 
3a). The TV set 30 is provided with means that emit an 
infrared beacon. The beacon allows the user equipment 
to detect the position of the TV set 30 relative to the field 
of view of the user. 

[0051 ] in Figure 3b the user selects the TV set 30 for 
interaction. Subsequent to the selection the system may 
display virtual icons 22, 22* that each represent a pos- 
sible action the user may take. That is, in the example 
the user may adjust the volume of the TV set or switch 
off the TV set. 

[0052] In Figure 3c the user 1 has moved his head so 
that the TV set 30 appears to be behind the desired ac- 
tion drop area. In the example control action object "tum 
off 22' is selected. After a predefined time has expired 
after the alignment of the TV set 30 and the action object 
22 the con^esponding action is perfomned i.e. the TV set 
is switched off. The action may be initiated by a control 
entity of the user equipment which may generate and 
transmit a control instruction signal to the TV set. 
[0053] Figure 3d shows a situation wherein the select- 
ed icon has a stronger appearance in order to provide 
the user with an indication that the selection has been 
done and is accepted. It may be advantageous in some 
application that the user needs first e.g. to release a but- 
ton or othenvise indicate that he/she accepts the action 
to be performed. 

[0054] It should be appreciated that whilst embodi- 
ments of the present invention have been described 
above in relation to head mounted displays, embodi- 
ments of the present invention are appibable to any oth- 
er suitable type of display equipment. For example, a 
hand-held display such as a display screen of a camera 
can be provided with the above described functionality. 
Such a device may conceptually resemble a hand-held 
magnifying class which allows the user to see real worid 
objects by looking through the display screen of the 
hand held device. 

[0055] Figure 5 shows a handheld device 50 provided 
with a transparent display 1 1 . An real worid object 20 (e. 
g. a household appliance) is located on top of a table 
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51. The display window 11 Is positioned such that the 
object 20 is visible in the selection area 21. The user 
may then select the object 20 by pressing a control but- 
ton 6. After.the selection the display window 1 1 is moved 
such that the object becomes aligned with one of the 
action activation objects 22. 
[0056] The handheld device 50 may comprise, for ex- 
ample, a mobile station. The mobile station may be pro- 
vided with a data processor facility 2 and data storage 
means 4. The mobile station may also be provided with 
transceiver means 5 for enabling communication via a 
wireless interface with another station. The other station 
may comprise a station provided in association with the 
real worid object or e.g. a base station of a communica* 
tion system. 

[0057] It is also possible to control the location and/or 
appearance of the selection area and the action drop 
areas. For example, it may be advantageous to move 
the virtual objects closer to each other when the real 
world object is substantially far away. By means of this 
it is possible to control the length of movement of the 
view that is required to bring the real worid object into 
association with a virtual object. 
[0056] It is also noted that instead of aligning the real 
worid object and the displayed virtual object, it is also 
possible to provide other type of visible association be- 
tween the objects. For example, the user may move the 
field of view such that the real word object and the virtual 
object can be seen to be located within a predefined ar- 
ea, e.g. in one comer or side of the view. 
[0059] The proposed augmented reality arrangement 
allows the eyes of a user to be focused to the selected 
real-world object at the same time whet the user Is se- 
lecting a desired action. Thus a strong visual binding 
may be provided between the object and the selected 
action. In other words, the user is able to see that the 
object and the desired action actually become associ- 
ated in his/her field of view. The user does not neces- 
sarily need to perfomr) any pointing gestures, press any 
buttons or give any voice commands. The user may con- 
trol remotely with any object that is provided with appro- 
priate means for enabling control thereof by means of 
the above described system. 
[0060] Processing associated with the recognition 
and/or selection of the object and/or the detection of the 
association of the object with the virtual object does not 
necessarily require much processing capacity. This is 
so since the image area to be analysed is substantially 
small. 

[0061] It is noted herein that while the above de- 
scribes exemplifying embodiments of the invention, 
there are several variations and modifications which 
may be made to the disclosed solution without departing 
from the scope of the present invention as defined in the 
appended claims. 



Claims 

I . A method of providing a user interiace, the method 
comprising: 

5 

providing a user with a view; 
selecting an object that is visible in the view; 
displaying a virtual object for the user, said vir- 
tual object being associated with an action; and 
10 selecting the action by moving the view and the 

virtual object relative to the object such that the 
object and the virtual object become associated 
with each other. 

IS 2. A method as claimed in claim 1 , wherein the user 
loolcs the view through a see-through display 
nrteans. 

3. A method as claimed in claim 2, wherein the user 
^ is provided with head mounted display means. 

4. A method as claimed in claim 2, wherein the user 
is provided with hand-held display means. 

25 5. A method as claimed In any preceding claim, where- 
in said selection of the object comprises a step of 
positioning the object within a selection area of the 
view. 

30 6. A method as claimed in claim 5, wherein the object 
is selected automatically after the object has been 
held in said selection area for a predefined period 
of time. 

35 7. A method as claimed in claim 5, wherein the selec- 
tion of the object in said selection area Is triggered 
by the user. 

8. A method as claimed in any preceding claim, where- 
to In the object sends a signal. 

9. A method as claimed in claim 8, wherein the object 
is recognised based on the signal. 

<5 10. A method as claimed in claim 8 or 9, wherein the 
object is Identified based on the signal. 

I I . A method as claimed in any of claims 8 to 1 0, where- 
in the signal is transmitted via an infrared link or a 

50 short range radio link. 

12. A method as claimed in any preceding claim, where- 
in the object is recognised by a camera means. 

55 13. A method as claimed in claim 12, wherein the rec- 
ognition and/or identification of the object is based 
on pattern or shape recognition. 
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14. A method as claimed in claim 13, wherein the pat- 
tem comprises a barcode. 

1 5. A method as claimed In any preceding claim, where- 
in the virtual object comprises an area in the view. 

16. A method as claimed in any preceding claim, where- 
in the association between the object and the virtual 
object is provided by aligning the objects with each 
other. 

1 7. A method as claimed in any preceding claim, where- 
in said action is selected after the association be- 
tween the object and the virtual object has been 
maintained for a predefined period of time. 

1 8. A method as claimed in any preceding claim, where- 
in the user conf inms that the object shall be subject- 
ed to an action that is indicated by said association 
between the object and the virtual object. 

1 9. A method as claimed in any preceding claim, where- 
in the object is subjected to a control operation in 
response to said selection of action. 

20. A method as claimed in any preceding claim, where- 
in infonmation associated with at least one possible 
action is communicated between the object and a 
control entity of the display means. 

21. A method as claimed in claim 20, wherein the com- 
munication occurs via a short range radio iinlc. 

22. A method as claimed in any preceding claim, where- 
in infomiation associated with at least one possible 
action is stored in a control entity of the display 
means. 

23. A method as claimed in any preceding claim, where- 
in infonnatlon associated with at least one possible 
action is communication via a data networl(to a con- 
trol entity of the display means. 

24. A user interface comprising display means, the dis- 
play means being adapted for displaying a virtual 
object for a user and enabling the user to see a real 
world object through the display means, wherein 
the user is enabled Interact with the object by mov- 
ing the display means relative to the real world ob- 
ject such that said virtual object is associated with 
the real world object. 

25. A user interface as claimed in claim 24, wherein the 
display means comprises a head mounted display. 

26. A user Interface as claimed in claim 24, wherein the 
display means comprises a hand-held display. 



27. A user interface as claimed in any of claims 24 to 
26, wherein the display means is adapted to display 
a selection area. 

s 28. A user Interface as claimed in claim 27 adapted to 
select the real world object automatlcally after the 
real world object has been held In said selection ar- 
ea for a predefined period of time. 

10 29. A user interface as claimed in any of claims 24 to 

29, comprising means for receiving a signal from 
the real world object. 

30. A user Interface as claimed in any of claims 24 to 
IS 29, comprising a camera means for detection of the 

real world object. 

31. A user interface as claimed In any of claims 24 to 

30, wherein the virtual object comprises an area in 
20 the field of view of the display means. 

32. A user interface as claimed in any of clalnns 24 to 

31 , wherein the association between the real world 
object and the virtual object is provided by aligning 

25 said objects with each other. 

33. A user interface as claimed in any of claims 24 to 
32 adapted to select an action after the real world 
object and the virtual object have been associated 

30 with each other for a predefined period of time. 

34. A user interface as claimed in any of claims 24 to 
33, comprising means for receiving infomiation as- 
sociated with at least one possible action from the 

35 real world object. 
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Fig. 2a Fig. 2b 
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Fig. 3a Fig. 3b 
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Look through a transparent display device 

Select a real world object from the field of view 
provided by the display device 

I 

Display at least one virtual object in the field of view, 
said virtual object associating with a control action 

i 

Move the display device such that the selected object 
becomes aligned with the virtual object 

Generate a control instruction signal 
Accomplish the indicated control action 



Fig- 4 
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